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PARTNERSHIP 

This discussion will recount some historical observations about establishing partnerships with the 
customer. It will suggest that such partnerships are established as the natural evolutionary product 
of a continuous improvement culture. Those are warm, ethereal terms about a topic that some 
people think already suffers from an excess of hot air. We will focus on some real-world 
activities and workplace artifacts to show there are substantive concepts behind the TQM 
buzzwords. 

PARAMAX MISSION 


Paramax sustains and maintains the software and data systems for ground-based operations at 
Johnson Space Center. We are a member of the (RSOC) Rockwell Space Operations Company 
(RSOC) Team that provides Space Shuttle and Space Station operations for the Mission 
Operations Directorate. 

Johnson Space Center owns a huge body of software (nearly 20 million lines of code) operating in 
a diverse mainframe, minicomputer and work station environments. The Paramax Mission is to 
ensure the operational suitability and readiness of this software baseline to meet the specific 
mission objectives of the Space Shuttle Program. 

We maintain the software tools used for mission planning, which starts years before a launch, and 
those used to manage the data that define flight-specific reconfiguration requirements, as well as 
the real-time systems that simulate operational conditions for pre-mission testing and training. We 
are also accountable for the real-time software that provides command, control and 
communications capabilities in the Mission Control Center. 

Maintaining the suitability and readiness of this operational software baseline for manned 
spaceflight missions is a sobering responsibility. Process management and continuous process 
improvement are essential to provide safe and reliable software services. Systematic and 
methodical approaches, with everyone's full and proactive participation, must be the norm. 
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GEORGE M. LOW 


We have found that the George M. Low benchmarking process provides a comparison to the best 
practices used in the NASA contractor community. The judging process, which compares our 
practices systematically against standard criteria used industry-wide, allows us to find our areas of 
strength and weakness. 

The systematic and objective characteristics of this evaluation process help us bring order to what 
can be a confusing as well as dynamic business. The resulting performance critique gives us 
feedback to provide lessons learned and establish improvement plans. Each year we have 
participated in the George M. Low award evaluation regimen, we have made significant 
improvements to our total performance as a direct result. 

EXCELLENCE TEAMS 

Our line organizations mesh directly with a set of formally established excellence teams that have 
a mandate to take ownership of a set of processes and focus on their improvement. Each team has 
a leader who acts as a coach or mentor Jo the team members. Every team member is expected to 
help define and improve processes, remove barriers that cause process problems, and in accepting 
accountability for the team's results. 

Each Team maintains its own records, analyzes its own processes, implements improvements and 
interacts with management and other teams to define and resolve issues. Management and non- 
management employees participate in these teams. Management is responsible for the teams' 
success, but each team has a large degree of autonomy in setting its own agenda. 

The teams communicate internally and externally by posting relevant information on bulletin 
boards called Excellence Boards. These boards identify the teams and display their respective 
priorities. Mission statements, metrics trends, corrective actions and other information relevant to 
the teams' activities and personalities are displayed as well. 

We have some examples of these boards in our booth at this symposium. If you go by and take a 
look, you will see that a great deal of pride and creative energy has gone into the information 
displays, along with an overriding sense of utility and functionality. Excellence Boards are not 
departmental bulletin boards; they are working tools. The boards are structured and controlled 
communication vehicles that provide visibility for our continuous improvement efforts. 

TQM IMPLEMENTATION 

Over the six years we have been at Johnson Space Center, we have found that our TQM 
implementation has produced a strange dichotomy. As our TQM involvement intensified, two 
classes of activities emerged. There was what many referred to as the "Quality Stuff," which 
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seemed to consist of meetings, discussions, newsletters, goal-setting sessions and great periods of 
introspection. There also was what was often termed "real work,” which consisted of people 
working to avoid errors, automating manual operations and generally advocating efficiency. 

We found, in fact, that two camps evolved: the one doing "Quality Stuff' and the one doing Real 
Work." The management team became agitated by this development and spent a great deal of 
energy worrying how to get the "Quality Stuff' people working together with the "Real Work" 
people to apply TQM in managing the business, 

BOTH CAMPS 

Eventually, both camps discovered that each was essential for a successful enterprise. We realized 
that some of our improvement activities were focused on what might be called "operational" 
things. These activities were aimed at classic process improvement and greatly emphasized doing 
things right the first time. People engaged in these activities believed they were doing the "real 
work." 

Other activities were focused on what we have come to recognize as "enabling" things. Their 
intent was to remove barriers and make sure effort was spent doing the right things. People 
engaged in these activities were often referred to as doing "Quality Stuff." 

The realization that both of these activities were part of the same thing was a major 
consciousness-raising milestone. It resulted in questions being asked that previously would not 
have been considered useful by people with the right information to pursue them. Doing things 
right and doing the right things are the inside and outside of the same circle. 

METRICS 

As we learned to improve processes and remove barriers, we also learned that appropriate metrics 
can support coherent goal-setting. By establishing program-level goals in the areas of quality, 
productivity, work flow and team-building, we were able to focus on processes and potential 
barriers at all levels within the organization. The program-level goals translated at all 
organizational levels into common metrics that aggregate naturally to provide a vertical metrics 
roll-up. Everyone's contribution to the top-level goals is measurable and actionable. 

METRICS AGGREGATION 

While vertical metrics aggregation is a matter of communicating goals at all levels in the 
organization, Metrics are aggregated horizontally by using the Oregon Matrix. This allows us to 
develop a performance index from a set of measurements that are mathematically unrelated. Each 
measurement is assigned a weight, and its matrix value is aggregated into a weighted average 
score. The matrix is calibrated periodically by setting current measurement trends at the matrix 


A3. 1-3 


value "3" level, the current goal at the "10" level and the unacceptable limit at the "0" level. 
Measurements are taken periodically, and trends for the matrix scores are calculated. There are 
many benefits in aggregating metrics this way. They help us understand complex relationships 
between seemingly independent variables, discern major contributions to problems or successes, 
and develop an overall sense of performance levels. 


COMPOSITE TRACK 


Composite averages, which we measured on our program-level metrics during the twelve months 
ending each July, show a flat trend through December, followed by a sudden drop and a steady 
improvement since. 


Each December we analyze our metrics process, compare it to our goals and make adjustments as 
appropriate. Last December we found that some of our raw measurements were not good 
indicators of the factors they were meant to convey, and we found that other measurements were 
calibrated in such a way that changes did not appear in the composite index. We rebaselined our 
measurement definitions and recalibrated the matrices, and the scores (not performance) dropped 
as a result and then trended upward, reflecting improvements implemented in the spring. 

PROBLEM PREVENTION 


The goal of continuous improvement is to move from reactively solving problems to proactively 
preventing them. To people in the software business, a major questions is: "How heavily is the 
customer affected by errors in the software?" Our goal is to prevent software errors from 
appearing in the operational baseline. We use a defect-density measurement (Software defects per 
million lines of code in the operational baseline) to track our performance. 

This chart shows our defect-density since we started handling Shuttle software. You can see that 
the trend is in the right direction. Some people believe such density reductions are the natural 
result of the software's maturation, and there is some truth in this. If software is static, eventually 
all the defects can be found and corrected. However this baseline has not been static. It grew by 
almost 50 percent during the past six years and huge amounts of changes have been made in it. 

We keep defect-density data normalized to the quantity of software that has been changed, as well 
as to total software, and both measurements trend downward. 

CUSTOMER RESPONSIVENESS 

Metrics also support customer responsiveness. The length of dwell time required to resolve 
critical problems and the backlog of unresolved discrepancy reports are just two areas where our 
customer's expectations are highly visible. These are global measurements that are part of the top- 
level metric trends that tell us how we are doing. 
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TQM SEASONS 


After pursuing continuous improvement for several years, we made another unexpected 
discovery. We went through seasons of awareness and opportunity much like annual seasons 
when certain conditions prevail and provide windows of opportunity. But unlike annual seasons, 
we found that as the TQM years cycled through their seasons, there was not an automatic 
progression. We found it was possible to get stuck, and no amount of process improvement and 
enabling activity would get us through that season until its major event had occurred. Imagine 
winter lasting until everyone has decorated a Christmas tree, or Fall until everyone had carved a 
pumpkin. 

We began in a happy Summer of process optimization. Everyone had a favorite process 
improvement, there was tremendous enthusiasm and it seemed possible to dream of something 
almost like spontaneous combustion or perpetual motion, continuous improvement directly to the 
fabled field of zero defects. This was how we began. 

Then we noticed the leaves turning brown. At first we reacted like a September butterfly, 
flapping our wings, making gaudy and colorful displays and trying to go on as before. However, 
we suspected things would not remain the same. Gradually, Management began to suspect that 
TQM was trickier than it looked. Behavioral change by our employees would not suffice; we 
would have to change as well. 

We launched with renewed enthusiasm into an Autumn of management adaptation. We found 
that management had a role to play in continuous improvement that was different and much more 
difficult: it required more listening and leadership. This challenged the management team to 
redefine a value-added role in a world whose process owners could change our own way of doing 
business. After a while, we began to see that management could become an organic part of 
continuous improvement, and we felt much better. 

Then the pond froze over. We suddenly entered a Winter of uncertainty as an unstated concern 
began to crystalize. The concern finally was expressed with the comment, "That's how we should 
do it, but the customer will never go for it." There was a widely held belief that process 
improvement was limited and that certain inefficiencies and absurdities could not be overcome 
because the customer demanded them. Voicing this concern enabled us to solve it, and we began - 
- at all levels -- to work with the customer on things perceived as issues. The excellence teams in 
particular lead the way in customer involvement, establishing well-understood expectations and 
means to measure our performance in meeting them. 

Then another surprising thing happened. Customer involvement lead to a springtime of renewed 
employee acceptance. Activities that had been considered extraneous or theoretical suddenly 
became bona fide customer requirements and, therefore, "real work." This renewed buy-in lead to 
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another Summer of process improvement and further evolution of the management team, to more 
sophisticated customer alignments, etc. 

As New Englanders have always known, the key to the cycle was Winter. Customer focus was 
the essential ingredient to permit continued progress. 

CUSTOMER INVOLVEMENT 


Customer Involvement is not a buzzword. It is a definable process that can be formalized as a 
series of activities. The most important step is the first: customers must be identified. Our 
excellence teams are responsible for displaying, on their excellence boards, formal identification 
of their internal and external customers. This is by no means a trivial task. 

The second step is to document a set of customer expectations that can be mutually understood by 
team members and customers. Getting customers to concur in writing is important, and displaying 
that agreement on excellence boards is essential. 

Once expectations have been established, a process to assess them must similarly be accepted 
mutually. This ensures that teams and customers periodically review and agree on how well 
expectations are being met. 

A customer score card is an important refinement of the assessment process. It allows metrics to 
be established that objectively depict the team's conformance to expectations. 

Finally, the score card process should be made part of the normal performance metrics process. 
This ensures that meeting customer expectations is not deemed just another set of performance 
criteria, but are Jjje criteria. Our excellence teams do this by developing Oregon Matrices of 
customer-related parameters, and incorporating these into the assessment and score card 
processes, 

INVOLVEMENT PROCESS 

Since customer involvement is a process, it can be measured like any other. Our excellence teams 
are progressing through the customer involvement steps, and many teams have developed some 
very effective tools to ensure clear communications between team members and customers. 

Nearly all our excellence teams have formally identified their customers. In addition three-fourths 
of the teams have signed mutually accepted expectations with their customers, and approximately 
two-thirds have assessment processes and score cards in place. However, only forty percent of 
our teams have integrated their customer score cards into their normal metrics process, and our 
big challenge for next year is to increase this number dramatically. 
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CUSTOMER IDENTIFICATION 


There are some examples of customer involvement artifacts that our excellence teams have 
developed. The first example shows tools used to make the customer identification formal and 
visible. 

The first of these is from a team that provides definable products and services to a variety of 
internal and external customers. The second is from a team with a hierarchical structure organized 
around work functions. The third is a product-oriented list that goes a step further: expectation 
identification. 

EXPECTATIONS 

Expectations must be both written and accepted. The first example here is a "performance pact 
document. The second takes the form of a creed (the sign-off was on the next page, which is not 
shown here). The third again carries expectations further, and includes measurements and 
analyses. 

ASSESSMENT 

The assessment process must routinely include the customer. The first example shows monthly 
feedback from the customer on specific expectations. The second is a very objective cooking 
survey. The last incorporates a score and the weighting of different assessment criteria. 

SCORE CARD 

The customer score card makes periodic assessments object and visible. The first example 
establishes success criteria and displays the customer's grade on each. The second is very similar. 
The third provides an aggregate or overall score that can be entered into a metrics process. 

INTEGRATED METRICS 

Finally, customer assessments should be integrated into the normal metrics process. The first 
example of this is a standard Oregon Matrix showing the customer's assessment, along with 
objective criteria important to the customer. The second illustration is a trend chart showing the 
Oregon Matrix index over time. The third is an example of a specific metric related to a customer 
expectation on rework that shows how the customer assessment process relates directly to 
preventive action and process improvement. 

IN CONCLUSION 

There really is no conclusion, since the only thing after continuous improvement is another 
opportunity. There are some observations, however. First, TQM is not a miracle; it is a change, 
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and change brings confusion to any organization. Management's role is more important than ever 
in dealing with this confusion, but the role is very different. 

Success with TQM is uncertain. It can be done incorrectly. An overemphasis on either the "real 
work" or "quality stuff without recognizing the importance of both will produce either highly 
centralized, authorization organizations which pretend TQM is enlightenment, or loose 
confederations of mobs and a management team in abdication. 

But, the benefits are worth the risk. The guiding principle is a simple definition of success: a joint 
venture with customers. 


